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(54) Proactive imaging system 

(57) A proactive imaging system includes proactive 
image having image data and an associated software 
agent for determining operations to be applied to the dig- 
ital image data, the operations including image process- 
ing and communicating the digital image data, and for 



initiating the determined operations; a communication 
network; a client computer connected to the communi- 
cation network for activating the software agent in the 
proactive image; and a server connected to the commu- 
nication network, the server including means for 
processing displaying or storing the digital image. 
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Description 

[0001] The present invention relates to image 
processing, communication, and display systems. In 
particular such systems that are employed to perform s 
digital photofinishing services. 
[0002] The beginning of an imaging chain generally 
includes an image capture device such as a camera or 
scanner. The camera may be a film camera that con- 
tains a light sensitive photographic emulsion and stores io 
the image as a latent image, or an electronic camera 
that contains a light sensor and stores the image in elec- 
tronic memory. The end of the chain, which may be re- 
ferred to as image output, generally includes a display 
device such as a printer for making a permanent version is 
of the image, or soft display device, such as a computer 
monitor or television screen. 

[0003] In between image capture and image output it 
is desirable to process the image. This is because the 
captured image is often not suitable for the output stage. 20 
For example, the captured image may have a large var- 
iation in the light level of bright and dark regions. If the 
image is printed, due to limitations of print media, the 
image detail in either the dark or bright regions will be 
lost. Another example is that cameras often produce an 25 
image defect that is commonly referred to as redeye. 
Redeye occurs when the pupils of people in the image 
appear red due to the reflection of light from the flash 
unit off of the retina. This results in a highly objectionable 
appearance of faces in a printed or displayed image. 30 
There is a multitude of other image processing that may 
be applied to an image including but not limit to the fol- 
lowing: detection of images that are duds so that media 
is not wasted by printing them; the detection of excellent 
images that should be printed multiple times, enlarged, 35 
placed in an album; increasing the sharpness of the im- 
age; cropping the image to emphasize the main subject; 
removing the affect of dust and scratches from the im- 
age; and improving the color of the image. 
[0004] The current method of image processing is to 40 
represent an image as passive data that is input to a 
chain of image processing routines. Recently, meta-da- 
ta has been included with the image. The meta-data 
consists of additional information such as subject dis- 
tance, light level, and whether the flash fired. One use 45 
of meta-data is to increase the efficiency of image 
processing. 

[0005] : A problem arises however because there is an 
ever increasing number of possible image processing 
and image understanding algorithms that can be applied so 
to an image to improve the quality and usage of the im- 
age. Furthermore, the number of images that are cap- 
tured and require processing is immense and growing. 
The problem with the current system is the reliance on 
a centralized image processing system to process every 55 
single image in the system in order to determine if some 
action needs to be taken to improve the quality or opti- 
mally use the image. This results in an un acceptably 



slow system throughput or requires very expensive 
computational means. 

[0006] U. S. Patent No. 5,913,088, issued June 15, 
1999, to Moghadam et al., entitled :" Photographic Sys- 
tem Capable of Creating and Utilizing Applets on Pho- 
tographic Film" discloses a photographic system capa- 
ble of creating and utilizing applets on photographic film. 
The applets, or a link to the applets are stored on a mag- 
netic layer of the film, and are programs that can be use 
to process a digital image derived by scanning an image 
on the film. Although this is concept useful, it is limited 
by the fact that the applet is not self executing. Specific 
instructions need to be provided by an operator to an 
image processor to apply the applet to the digital image 
data. There is a need therefore for an improved photo- 
graphic system that does not require operator interven- 
tion. 

[0007] U. S. Patent No. 5,974,201 , issued October 26, 
1 999 to Chang et al., entitled "Smart Image System 0 dis- 
closes a smart image system that provides image data 
with a knowledge structure consisting of protocols, hot 
spots, active indexes, and attributes. The smart images 
automatically respond to environmental changes. This 
concept is useful, but the smart image system does not 
provide a means for the image to be processed in the 
sense that pixel values are modified. Furthermore, it 
does not provide a means for automatically determining 
image content without input from a human. 
[0008] The need is met according to the present in- 
vention by providing an imaging system in which image 
data is combined with a proactive self executing agent 
that determines what processing is necessary, and au- 
tomatically obtains the necessary processing. Instead 
of an image being passive data it becomes an active 
component that has the means of determining the op- 
erations such as processing storing, communication 
and printing that is required and the means to obtain the 
necessary processing. 

[0009] According to one aspect of the invention, a 
proactive imaging system includes a proactive image 
having image data and an associated software agent for 
determining operations to be applied to the image data, 
the operations including image processing and commu- 
nicating the digital image data, and for initiating the de- 
termined operations; a communication network; a client 
computer connected to the communication network for 
activatingthe software agent in the proactive image; and 
a server connected to the communication network, the 
server including means for processing displaying or 
storing the digital image. 

[0010] In one embodiment, the image is a digital im- 
age and the software agent is a computer program as- 
sociated with the image, either directly, or by means of 
a reference to the location of the program on a commu- 
nication network. According to an alternative embodi- 
ment, the image is a hard copy image, and the software 
agent is associated with the image by means of a coded 
message on the image indicating the location of the 
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computer program on a communication network. 
[001 1 J The proactive imaging system according to the 
present invention has the advantage that no human in- 
tervention is needed to determine and effect the opera- 
tions that are performed on the digital image data once 
the image component is created and activated. 

Fig. 1 is block diagram showing the proactive imag- 
ing system according to the present invention; 
Fig. 2 is a data structure showing one embodiment 
of a proactive image including an agent according 
to the present invention; and 
Fig. 3 is a block diagram showing the structure of 
the agent shown in Fig. 2. 

[0012] A proactive imaging system according to the 
present invention is shown in Fig. 1 . A camera 1 0 cap- 
tures an image along with metadata that contains infor- 
mation on the capture conditions such as lens focal 
length, brightness, subject distance, illuminant, f- 
number, whether the flash fired, time and place of image 
capture, and any other potentially useful information. 
According to one embodiment, the camera electronical- 
ly captures a digital image and the digital image is di- 
rectly transferred to a computer 12. it will be understood 
that computer 12 can be any apparatus with a micro- 
processor, such as a personal computer, set top box, or 
any other appliance containing a microprocessor with 
capability to handle digital images, such as communi- 
cation equipment, including cellular phones, personal 
digital assistants (PDA's), or appliances such as televi- 
sion receivers, refrigerators, stereo equipment, or game 
consoles, or transportation vehicles equipped with tele- 
communication capability. According to another embod- 
iment?* the-camera captures the image using- photo- 
graphic film. The film is developed and scanned to pro- 
duce a digital image that is transferred to the computer 
12. The digital image data is associated with a software 
agent in the electronic camera to form the proactive im- 
age according to the present invention. The proactive 
image is an object in an object oriented programming 
system. Alternatively, the digital image may be associ- 
ated with the software agent in the computer 12 when 
the image data is transferred to the computer to form 
the proactive image. In the embodiment where the im- 
age is captured on photographic film, the proactive im- 
age, includes the latent image on the film plus a refer- 
ence to ther location of the software agent on a commu- 
nications network. The reference included with the film 
can be optically, magnetically, or photographically en- 
coded on the film using well known techniques for re- 
cording data on a film in a camera. 
[0013] Alternatively, the proactive image may be cre- 
ated by associating the software agent with the image 
after the developed film is scanned but before the digital 
image data is transferred to the computer 12 or after it 
is transferred to the computer by means of software 
stored on the computer. 



[0014] The proactive image may also be embodied as 
a hard copy image, such as a slide or photographic print 
with accompanying software agent location data. The 
location data can be stored on the print, for example by 

5 a magnetic layer on the print, a printed bar code on the 
back of the print, an infrared readable layer on the front 
or back of the print having software agent location data. 
Alternatively, the software agent location data can be 
embedded in the image on the print, using stegano- 

io graphic data embedding techniques. These techniques 
can also be used in a digital camera 10, or in the com- 
puter 12 to embed the location of the software agent di- 
rectly into the digital image. 

[0015] The present invention provides the proactive 

is image with the means to determine what actions are to 
be taken by the use of image understanding software 
algorithms that are part of the proactive digital image or 
are referenced by the proactive digital image. The image 
understanding algorithms may use the pixel code values 

20 jn the image the metadata, or both in order to determine 
actions to be taken. For example, the proactive digital 
image may determine that the image is of high quality 
and action should betaken to print multiple copies of the 
image data. The proactive images may also communi- 

25 cate with each other. For instance, if several proactive 
images are present on computer 12 inter-proactive im- 
age communication can determine which image is of the 
highest quality and only the highest quality image will 
act to print the image data. According to another em- 

30 bodiment, the proactive image may use the metadata 
associated with the image indicating whether the flash 
fired, the brightness when the image was captured, and 
the subject distance in order to determine the probability 
that a redeye defect exists in the image. If the probability 

35, -exceeds a predetermined threshold the proactive image- ■ 
takes the actions necessary to obtain redeye detection 
and correction. 

[0016] The computer 12 is connected by means of a 
network 1 4 to a server 1 6. The proactive image may ob- 

40 tain the address on the network of the server 16 from 
the camera 10, from data on the photographic film, or 
from data stored on the computer 12. The proactive im- 
age may transfer the image data to the server 16 over 
the network 14. Alternatively, the server 1 6 may transfer 

45 executable code to the computer 12 to be used by the 
proactive image. The server 1 6 may be instructed by the 
proactive image to display, 18, print 20, process 22, or 
store 24 the image data. ~" " l *" 
[0017] Referring in Fig. 2 a proactive image, generally 

so designated 25 consists of a software agent 26, metada- 
ta 28 and image data 30. The metadata 28 may include 
information on image capture conditions, place, time, 
and other information that may be useful such as the 
address of an image processing server that may be 

55 used by the software agent 26. The metadata portion of 
the proactive image may be omitted if the camera is not 
capable of supplying metadata. The image data 30 is 
the raw pixel code values plus other information such 
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as image size and colorspace. This may also be a link 
to the image data. 

[0018] The software agent 26 exhibits three proper- 
ties that are characteristic of an agent. It can cooperate, 
learn, and is autonomous. This is described in detail re- 
ferring to Fig. 3. The functions of the agent 26 is divided 
into three units. The communication unit 32 is able to 
contact, exchange data with, and invoke actions of serv- 
ers over a network and other agents. This unit may use 
the Internet Inter-ORB Protocol (HOP), Remote Method 
Invocation (RMI), Distributed Component Object Model 
(DCOM), or any other standard for communication be- 
tween distributed objects. The communication unit 
along with the other units allows the agent 26 to coop- 
erate with other agents and other software objects. The 
image understanding unit 34 is able to learn the char- 
acteristics of the image data using methods of pattern 
recognition. The image understanding unit 34 may also 
use the metadata to aid the pattern recognition process. 
The rules unit 36 is responsible for autonomous action. 
It contains a rule system or other means of artificial in- 
telligence that, based on input from the image under- 
standing unit 34, determines the actions that need to be 
taken. An action that requires interaction with a server, 
other agents, or any other software object is taken 
through means of the communication unit 32. 



Claims 

1 . A proactive imaging system, comprising 

a) a proactive image including image data and 
an associated software agent for determining 

> ~ ^ operations*© be applied to trie-digital image da- 
ta, including image processing and communi- 
cating the digital image data, and for initiating 
the determined operations; 

b) a communication network; 

c) a client computer connected to the commu- 
nication network for activating the software 
agent in the proactive image; and 

d) a server connected to the communication 
network, the server including means for 
processing displaying or storing the digital im- 
age. 

2. The proactive imaging- system claimed in claim 1 , 
further comprising: an electronic camera including 
means for creating a digital image and means for 
associating a software agent with the digital image 
to form the proactive image. 

3. The proactive imaging system claimed in claim 2, 
wherein the software agent is associated with the 
image by appending a software program embody- 
ing the agent to the digital image. 



4. The proactive imaging system claimed in claim 2, 
wherein the software agent is associated with the 
image by appending a reference to the communi- 
cation network accessible location of a software 

5 program embodying the agent to the digital image. 

5. The proactive imaging system claimed in daim 2, 
wherein the software agent is associated with the 
image by embedding a reference to the communi- 

10 cation network accessible location of a software 
program embodying the agent in the digital image. 

6. The proactive imaging system claimed in claim 1 , 
further comprising: an electronic camera including 

15 means for creating a digital image and wherein the 
client computer includes means for associating the 
software agent with the digital image to form the 
proactive image. 

20 7. The proactive imaging system claimed in claim 6, 
wherein the software agent is associated with the 
image by appending a software program embody- 
ing the agent to the digital image. 

25 8. The proactive imaging system claimed in claim 6, 
wherein the software agent is associated with the 
image by appending a reference to the location of 
a software program embodying the agent to the dig- 
ital image. 

30 

9. The proactive imaging system claimed in daim 6, 
wherein the software agent is associated with the 
image by embedding a reference to the location of 
a software program embodying the agent in the dig- 

*ital image; - ^,- t -^.™-r„„^«..: — « — — 

10. The proactive imaging system claimed in daim 1 , 
further comprising a photographic camera for cap- 
turing an image on film, the camera induding 

40 means for associating the software agent with the 
image on the film. 

11. The proactive imaging system claimed in claim 10, 
wherein the means for associating the software 

45 agent comprises means for recording a reference 
to the location of a software program embodying the 
agent : .. ..^_ r 

12. The proactive imaging system claimed in claim 11 , 
so wherein the means for associating the software 

agent is a modulatable light source and the refer- 
ence is recorded optically on the film. 

13. The proactive imaging system claimed in claim 11 , 
55 wherein the means for associating the software 

agent is a magnetic recording head and the refer- 
ence is recorded magnetically on a magnetic layer 
on the film. 
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14. An electronic camera comprising an image sensor 
for creating a digital image, and means for associ- 
ating a software agent with the digital image for de- 
termining operations to be applied to the digital im- 
age data, including image processing and commu- s 
nicating the digital image data. 

15. The electronic camera claimed in claim 14, further 
comprising means for associating metadata relat- 
ing to image capture conditions with the digital im- io 
age. 

16. A proactive image, comprising: image data and a 
software agent associated with the image data for 
determining operations to be applied to the digital 15 
image data, including image processing and com- 
municating the digital image data. 

17. The proactive image claimed in claim 16, further 
comprising metadata relating to image capture con- 20 
ditions. 
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